Electronic system with palm recognition, vehicle and method for operating the same

ABSTRACT

A method of operating an electronic system for is provided. The method includes the following actions. A plurality of images are captured by an image capturing unit. An object is detected by a sensor unit within a sensing region. When the object is detected within the sensing region of the sensor unit, a palm feature is recognized by a processing unit from the captured images. An application is stimulated in response to the palm feature.

CROSS REFERENCE

This application claims the benefit and priority to of U.S. ProvisionalApplication Ser. No. 62/757,744, filed on Nov. 8, 2018, and entitled“CENTRAL CONSOLE, INFOTAINMENT SYSTEM AND OPERATING METHOD WITH A PALMRECOGNITION”, which is incorporated herein by reference in its entirety.

FIELD

The present disclosure generally relates to an electronic system withpalm recognition, a method for operating the same, and a vehicle withpalm recognition.

BACKGROUND

In a vehicle, an infotainment system is embedded in the middle of thecar frame between the driver and the passenger. When operating theinfotainment system, the driver and the passenger have to reach his orher arm to touch the buttons, control panel or touch screen of theinfotainment system. On top of that, when the driver operates theinfotainment system, the driver looks away from the road to check theinfotainment system, and this kind of distraction puts the driver andpassenger(s) at a high risk of a crash. Therefore, it is desirable toprovide a convenient and safer infotainment system.

SUMMARY

In one aspect of the present disclosure, a method for operating anelectronic system is provided. The method includes the followingactions. A plurality of images are captured by an image capturing unit.An object is detected by a sensor unit within a sensing region. When theobject is detected within the sensing region of the sensor unit, a palmfeature is recognized by a processing unit from the captured images. Anapplication is stimulated in response to the palm feature.

In another aspect of the present disclosure, an electronic system isprovided. The electronic system includes an image capturing unit, asensor unit, and a processing unit. The image capturing unit isconfigured to capture a plurality of images. The sensor unit isconfigured to detect an object within a sensing region. The processingunit is configured to recognize a palm feature from the captured imageswhen the object is detected within the sensing region of the sensorunit, and to stimulates an application in response to the palm feature.

In yet another aspect of the present disclosure, a vehicle is provided.The vehicle includes an infotainment system. The infotainment systemincludes an image capturing unit, a sensor unit, a display device, and aprocessing unit. The image capturing unit is configured to capture aplurality of images. The sensor is configured to detect an object withina sensing region. The processing unit is configured to recognize a palmfeature from the captured images when the object is detected within thesensing region of the sensor unit, and to display an application on thedisplay device in response to the palm feature.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an electronic system according to animplementation of the present disclosure.

FIG. 2 is a schematic diagram showing the front of the vehicle interioraccording to an implementation of the present disclosure.

FIG. 3 is a flowchart of the method for operating the electronic systemaccording to an embodiment of the present disclosure.

FIG. 4 is a schematic diagram of a captured image including the palmfeature according to an implementation of the present disclosure.

FIG. 5A is a schematic diagram illustrating an implementation of theoperation of the infotainment system according to the palm recognition.

FIG. 5B is a schematic diagram illustrating another implementation ofthe operation of the infotainment system according to the palmrecognition.

FIGS. 6A-6D are schematic diagrams of multiple positions of the displaydevice according to an implementation of the present disclosure.

FIG. 7 is a schematic diagram of one implementation of the operation ofthe infotainment system in response to the palm recognition.

DETAILED DESCRIPTION

The following description contains specific information pertaining toexemplary implementations in the present disclosure. The drawings in thepresent disclosure and their accompanying detailed description aredirected to merely exemplary implementations. However, the presentdisclosure is not limited to merely these exemplary implementations.Other variations and implementations of the present disclosure willoccur to those skilled in the art. Unless noted otherwise, like orcorresponding elements among the figures may be indicated by like orcorresponding reference numerals. Moreover, the drawings andillustrations in the present disclosure are generally not to scale, andare not intended to correspond to actual relative dimensions.

FIG. 1 is a block diagram of an electronic system 100 according to animplementation of the present disclosure. The electronic system 100includes an image capturing unit 114, a sensor unit 116 and a processingunit 118. The image capturing unit 114 is configured to capture imagesof a user. In one implementation, the image capturing unit 114 mayinclude, but not limited to, one or more cameras or any device capableof capturing images. The image capturing unit 114 may be a depth-sensingcamera with a depth sensor. The camera may be an RGB color camera or aninfrared (IR) camera. In some embodiments, the image capturing unit 114further includes a light source (e.g., an IR projector or a visiblelight projector) enabling instant profiling of the hand/palm of theuser. With the light source and high dynamic range (HDR) imaging, thepalm recognition may be adapted to a darker environment.

The sensor unit 116 is configured to detect an object within a sensingregion. In one implementation, the sensor unit 116 is a proximitysensor. In another implementation, the sensor unit 116 is realized by adepth sensor of a depth sensing camera.

The processing unit 118 is coupled to the image capturing unit 114 andthe sensor unit 116. The processing unit 118 may process data andinstructions. In one embodiment, the processing unit 118 may be ahardware module comprising one or more central processing unit (CPU),microcontroller(s), ASIC, or a combination of above but is not limitedthereof. The processing unit 118 performs image recognition and imageprocessing to render images. In one embodiment, the processing unit 118is configured to identify the captured images and perform palmrecognition on the captured images. In some embodiments, the processingunit 118 further performs facial recognition, voice recognition andother biometric recognition based on the captured images. In someembodiments, the processing unit 118 further provides driver monitoringfunction, such as, gaze zone prioritization, fatigue recognition,inattention recognition, and distraction recognition to the driver.

Optionally, the electronic system 100 further includes a display device112, coupled to the processing unit 118, configured to display imagesand applications. In one embodiment, the electronic system 100 furtherincludes a machinery configured to control a movement maneuver of thedisplay device 112.

In an exemplary implementation, the electronic system 100 is anin-vehicle infotainment system. In another implementation, theelectronic system 100 is a central console of the vehicle. In some otherimplementations, the electronic system 100 could be any electronicdevice disposed in the vehicle. FIG. 2 is a schematic diagram 200showing the front of the vehicle interior an implementation of thepresent disclosure. As shown in FIG. 2, a driver's seat 210 is locatedon the left side of the vehicle, while a front passenger seat 220 islocated on the right side. However, the driver seating position is notlimited on the left side, and may be disposed according to the rule ofthe road (left-hand traffic or right-hand traffic). A vehicle instrumentcluster or a dashboard 120 is arranged directly ahead of the driver'sseat 210. The dashboard 120 may include, but not limited to, aspeedometer showing a current speed of the vehicle in the middlesection, a tachometer showing a rotation speed of the engine of thevehicle on the left of the speedometer, and an odometer showing thedistance travelled by the vehicle on the right of the speedometer. Inone implementation, the dashboard 120 is a digital dashboard including adigital display panel.

In one embodiment, the infotainment system 110 locates in the center ofthe front of the vehicle interior as shown in FIG. 2, which issubstantially disposed between the driver's seat and the frontpassenger's seat of a vehicle, so that both the driver and the passengermay share the control of the infotainment system 110. In someembodiments, the infotainment system 110 locates in the center of theback seat so that the backseat passengers could share the control of theinfotainment system 110.

As stated above, the infotainment system 110 includes a display device112, an image capturing unit 114, a sensor unit 116 and a processingunit (not shown). In this implementation, the display device 112 isconfigured to display applications. The image capturing unit 114 isdisposed on the display device 112 to capture the images of thepassenger or the driver. In some other implementations, the imagecapturing unit 114 is not necessarily disposed on the display device 112but disposed in the vehicle. The sensor unit 116 is disposed on thedisplay device 112. In one embodiment, the processing unit 118 is one ofthe functional modules of an automotive electronic control unit (ECU).In some other embodiments, the processing unit 116 consolidates thehardware controller in a vehicle for controlling the cluster, in-vehicleinfotainment (IVI) system, head-up display (HUD), rear-view mirror, etc.

FIG. 3 is a flowchart of the method for operating the electronic system(e.g., the infotainment system 110) according to an embodiment of thepresent disclosure. The method includes the following actions. In action310, the image capturing unit 114 captures images.

In action 320, the sensor unit 116 detects an object within a sensingregion. For instance, a proximity sensor detects the presence of nearbyobjects. When a nearby object is detected within a certain range, thepalm recognition will be initiated. In other words, the palm recognitionis not performed when a hand is not detected around the display device112. In this case, the infotainment system 110 could stay in a standbymode when the hand is not detected, and therefore the power consumptioncould be reduced.

In action 330, the palm recognition is performed. That is, theprocessing unit 118 recognizes a palm feature from the captured images.For example, the user may enroll palm images in advance. Specifically,the processing unit 118 records and stores the palm feature of the palmimages associating with a registered user. In one embodiment, the palmfeature may include, but not limited to, a palm print, a fingerprint, ahand pattern, a hand size, a palm size, a finger size, a hand gesture, apalm gesture, a finger gesture a palm attribute or the combination ofthe above. And when the palm recognition is initiated. And then, theprocessing unit 118 identifies the palm feature according to, forinstance, the position, the pattern, the gesture of one or more parts ofa finger, palm and/or hand. In one embodiment, the palm recognitionincludes a hand gesture, a palm gesture and/or a finger gesturerecognition. For instance, the hand gesture may include, but not limitedto, pulling up, pushing down, calling over, grabbing, waving, pointing adirection. The palm gesture may include, but not limited to, a closinghand forming a fist, an opening hand, and other positions and gesturesby the palm. The finger gesture may include, but not limited to, athumbs-up, a thumbs-down, the number gestures, and other gestures bymoving or folding each finger. The palm attribute is of a right hand orof a left hand. The palm attribute could be identified from the palmimages according to all kinds of hand/palm characteristics.

In one embodiment, when the palm feature is recognized, the methodfurther includes the following actions. Firstly, the processing unit 118determines a bounding box of the captured images for tracking the palm.As shown in FIG. 4, a bounding box B_(p) including the palm region C_(p)is marked. Next, the palm region C_(p) is cropped for furtheridentification. Optionally, the palm region is calibrated to a preferredorientation, e.g. upright position, by the image signal processing (ISP)process. Afterwards, the palm feature (e.g., a palm print) is identifiedfrom the palm region. Finally, the identified palm feature is used forpalm recognition. In addition, the processing unit compares theidentified palm feature with the enrolled palm image(s) and recognize aregistered user if the result matches.

In action 340, the processing unit 118 stimulates an application inresponse to the identified palm feature. In one embodiment, each usercould pre-configure a default application, and after the palm print isrecognized, the processing unit 118 further identifies a user identityaccording to the palm print, and then provides the correspondingapplication. The corresponding application could be a personalizedapplications or functions for each user. In some embodiments, theapplication is displayed on a display device. For instance, a navigationmap is displayed on the display device 112 when the driver's palm isrecognized, while a music playing application is stimulated and/ordisplayed on the display device 112 when a passenger's palm isrecognized. Furthermore, personal content (e.g., such as personalmusic/video playlist, background setting, theme setting or thepersonalized desktop application) is stimulated and/or displayedaccording to the recognized palm print.

FIG. 5A is a schematic diagram illustrating an implementation of theoperation of the infotainment system 110 according to the palmrecognition. As shown in FIG. 5A, the infotainment system 110 displays apersonal content 562, e.g., a personal playlist, on the display deviceaccording to the palm print 454 of a user. It is noted that the palmprint 454 shown in FIG. 5A is for illustration purposes only; however,in practical situation that the palm recognition is implemented, inorder to capture the palm print, the palm needs to face toward the imagecapturing unit.

In another embodiment, the user could pre-configure one or more palmgestures to represent different applications, and when the specific palmgesture is recognized, the processing unit 118 displays an applicationon the display device 112 accordingly. For instance, a navigation map isdisplayed on the displayed device when a closed first is recognized,while a music playing application is displayed on the displayed devicewhen a victory sign gesture is recognized. That is, more gesture (staticor dynamic) could be set in advance such that various applications(e.g., navigation map, music playing application, video playingapplication, call list) could be initiated quickly.

Moreover, the user may control the infotainment system 110 via differentgestures. The user may use the hand gesture, the palm gesture and/or thefinger gesture to operate the infotainment system, e.g., switchingbetween the applications, selecting and/or performing a function. Forexample, the user could make a swiping right/left gesture to scroll theapplication list to the right/left and plays/selects the applications onthe left/right. In another example, the user could make a thumbs-up toenter an application or the next level of menu.

FIG. 5B is a schematic diagram illustrating another implementation ofthe operation of the infotainment system 110 according to the palmrecognition. In this embodiment, the infotainment system 110 displays acontent 560 on the display device according to the palm gesture 452.Specifically, the palm gesture 452 is a swiping right gesture, and thenthe displayed application list is scrolling to the right accordingly.

Based on the above, the user may operate the infotainment system 110remotely without touching the display device 112 of the infotainmentsystem 110. Since the physical contact to the device could be reduced,it is much safer for the driver to operate the system while driving. Insome embodiments, although the infotainment system 110 is currentlyoperated by a left seat user, the image capturing unit 114 keepsmonitoring the right seat user, and thus the right seat user may alsooperate the infotainment system at the same time. Therefore, both thedriver and the passenger could share the control of the infotainmentsystem easily without physically hanging over the infotainment system110.

In some embodiments, the display device 112 is movable. Specifically,the position and orientation of the display device 112 may be adjustedto move towards the driver or the passenger according to the user'soperation.

Since a conventional central console or infotainment system is fixed inthe car frame, the passenger and the driver might have to lean forwardor stretch arms to reach a control panel or screen. In contrast, thedisplay device 112 could be moved to a position near the passenger and aposition near the driver such that the driver or the passenger couldview the display content clearly. In another case that the physicalcontact to operate the device is required, the display device 112 couldbe moved to a hand-reachable position for the passenger or the driver tooperate the system without leaning forward or stretching arms to reachthe control panel or screen. In addition, the display device 112 couldbe moved a position near the dashboard, or in front of the driver, whichis in the peripheral vision of the driver when the driver keeps his/hereyes on the road. As such, the infotainment system 110 is designed tomake the operation more convenient for both the driver and thepassenger. Also, it is much safer for the driver to keep the distractionto its lowest while driving.

FIGS. 6A-6D are schematic diagrams of multiple positions of the displaydevice 112 according to an implementation of the present disclosure. Inthis implementation, the machinery for controlling the movement maneuverof the display device is a movable supporting bracket. However, thepresent disclosure is not limited thereto. The machinery may use anykinds of elements or structures to adjust the orientation of the displaydevice and move the position of the display device. As shown in FIG. 6A,initially, the display device 112 locates at a position P0, which isembedded in the car frame 630 via a movable supporting bracket (notshown). For instance, after the infotainment system 110 is turned on,the display device 112 moves to a position P1 by stretching thesupporting bracket b1 to the front as shown in FIG. 6B. In oneimplementation, the infotainment system 110 is turned on when a sensorunit detects that a hand/palm is within a sensing region. In anotherimplementation, the infotainment system 110 is turned on when a pullingup gesture made by a user is detected.

Afterwards, the display device 112 moves to the hand-reachable positionof the user. For instance, the infotainment system 110 rotates thesupporting bracket b1 to move the display device 112 to a position P2Ain front of a left seat user (e.g., the driver) and an orientationfacing towards the user on the left seat (e.g., the driver) as shown inFIG. 6C. Alternatively, as shown in FIG. 6D, the infotainment system 110rotates the supporting bracket b1 to move the display device 112 to aposition P2B in front of the right seat user (e.g., the passenger) andan orientation facing towards the user on the right seat (e.g., thepassenger).

The display device 112 may move to the left side for the left seat useror to the right side for the right seat user according to the palmrecognition. FIG. 7 is a schematic diagram of one implementation of theoperation of the infotainment system in response to the palmrecognition. In one embodiment, the processing unit 116 further obtainsa palm position of the identified palm based on the image captured bythe image capturing unit 114, and determines the target position of thedisplay device in response to the palm position. For example, when theprocessing unit 116 determines that the identified palm locates near theleft seat region (e.g., 452), the display device 112 moves to ahand-reachable position of the left seat user (e.g., P2A). Similarly,when the processing unit 116 determines that the identified palm locatesnear the right seat region (e.g., 454), the display device 112 moves toa hand-reachable position of the right seat user (e.g., P2B).

In another embodiment, the processing unit determines the movementmaneuver of the display device according to the palm attribute. Sincethe infotainment system 110 is on the right side of the left seat user,it is more natural for the left seat user to use the right hand tooperate the infotainment system 110; and therefore when the processingunit 116 identifies that the palm attribute is of a right hand (e.g.,442), the display device 112 moves to a hand-reachable position of theleft seat user (e.g., P2A). Similarly, when the processing unit 116identifies that the palm attribute is of a left hand (e.g., 444), thedisplay device 112 moves to a hand-reachable position of the right seatuser (e.g., P2B). Moreover, after the determination of the palmattribute is of a right hand or a left hand, different application couldbe displayed for the left seat user or the right seat user accordingly.

In some embodiments, although the display device 112 is at a positionnear a left seat user, the image capturing unit 114 keeps monitoring theright seat user, and thus the right seat user may also operate theinfotainment system 110 at the same time without touching theinfotainment system 110.

In some other embodiments, the movement maneuver of the display device112 is determined according to a palm gesture made by the user. Forinstance, the user could make a palm gesture by waving his/her palm(e.g., leftward, rightward, upward, downward, forward, backward, or anycombination of the above), and the processing unit 118 determines amovement maneuver of the display device 112 according to the palmgesture and controls the display device 112 to move accordingly.Specifically, the user could use palm gesture to move, tilt and/orrotate the display device 112, thereby adjusting the display device 112to the desirable position and/or the desirable orientation.

In some embodiments, when the driver is operating the infotainmentsystem 110, the processing unit 118 performs driver monitoring function,such as, gaze zone prioritization, fatigue recognition, inattentionrecognition, and distraction recognition to the driver. For instance,the image capturing unit 114 keeps capturing images of the drivers, andthe processing unit 118 monitors a state of a driver by tracking the eyegaze, the blinking, or the head pose of the driver or other facialfeatures. The state of the driver is further used for determining thedistraction level, response time and fatigue of the driver. The state ofthe driver may include, for example, sleepy or not, and concentrate onthe road or not. In one implementation, when the processing unit 116determines that the driver is distracted or tired (e.g., when thedistraction level exceeds a threshold), the display device 112 displaysa warning sign on the screen. In another implementation, when theprocessing unit 116 determines that the driver is distracted or tired,the display device 112 of the infotainment system 110 is turned off. Insome other implementations, even though the same application isperformed and displayed on the display device 112, when the driver isoperating the infotainment system 110, the driving information isdisplayed on a part of the screen. Therefore, the driver could interactsafely with the infotainment system 110 while driving.

Based on the above, several electronic systems and methods for operatingan electronic system are provided in the present disclosure. Theimplementations shown and described above are only examples. Even thoughnumerous characteristics and advantages of the present technology havebeen set forth in the foregoing description, together with details ofthe structure and function of the present disclosure, the disclosure isillustrative only, and changes may be made in the detail, including inmatters of shape, size and arrangement of the parts within theprinciples of the present disclosure up to, and including, the fullextent established by the broad general meaning of the terms used in theclaims.

What is claimed is:
 1. A method for operating an electronic system, comprising: capturing, by an image capturing unit, a plurality of images; detecting, by a sensor unit, an object within a sensing region; recognizing, by a processing unit, a palm feature from the captured images when the object is detected within the sensing region of the sensor unit; stimulating, by the processing unit, an application in response to the palm feature, wherein the palm feature includes a palm gesture; determining, by the processing unit, a movement maneuver of a display device according to directional information of the palm gesture; and moving, by the processing unit, the display device to a desirable position and a desirable orientation according to the movement maneuver.
 2. The method of claim 1, wherein the palm feature includes a palm print, and the step of recognizing the palm feature from the captured images further comprises: determining, by the processing unit, a bounding box including a palm region; and identifying, by the processing unit, the palm print from the bounding box including the palm region.
 3. The method of claim 1, wherein the palm feature includes a palm print, and the method further comprises: identifying, by the processing unit, a user identity according to the palm print; wherein the stimulated application corresponds to the user identity.
 4. The method of claim 1, wherein the palm feature includes a palm attribute, and the method further comprises: determining, by the processing unit, whether the palm attribute is of a right hand or a left hand; wherein a first application is performed when the palm attribute is of the right hand, and a second application is performed when the palm attribute is of the left hand.
 5. The method of claim 4, further comprising: determining, by the processing unit, the movement maneuver of a display device according to the palm attribute; wherein the display device is moved to a first orientation and a first position when the palm attribute is of the right hand, and the display device is moved to a second orientation and a second position when the palm attribute is of the left hand.
 6. The method of claim 5, wherein the electronic system is substantially disposed between a driver's seat and a front passenger's seat of a vehicle, and the first position is in front of a left seat of the vehicle, and the second position is in front of a right seat of the vehicle.
 7. The method of claim 3, wherein the stimulated application corresponds to the user identity is a personalized desktop application on a display device.
 8. An electronic system, comprising: an image capturing unit configured to capture a plurality of images; a sensor unit configured to detect an object within a sensing region; a display device; a machinery configured to control a movement maneuver of the display device; and a processing unit configured to perform instructions for: recognizing a palm feature from the captured images when the object is detected within the sensing region of the sensor unit; stimulating an application in response to the palm feature, wherein the palm feature includes a palm gesture; determining the movement maneuver of the display device according to directional information of the palm gesture; and moving the display device to a desirable position and a desirable orientation according to the movement maneuver.
 9. The electronic system of claim 8, wherein the palm feature includes a palm print, and when the palm feature is recognized, the processing unit is further configured to perform instructions for: determining a bounding box including a palm region; and identifying the palm print from the bounding box including the palm region.
 10. The electronic system of claim 8, wherein the palm feature includes a palm print, and the processing unit is further configured to perform instructions for: identifying a user identity according to the palm print; wherein the stimulated application corresponds to the user identity.
 11. The electronic system of claim 8, wherein the palm feature includes a palm attribute, and the processing unit is further configured to perform instructions for: determining whether the palm attribute is of a right hand or a left hand; wherein a first application is performed when the palm attribute is of the right hand, and a second application is performed when the palm attribute is of the left hand.
 12. The electronic system of claim 11, wherein the processing unit is further configured to perform instructions for: determining the movement maneuver of the display device according to the palm attribute; wherein the display device is moved to a first orientation and a first position when the palm attribute is of the right hand, and the display device is moved to a second orientation and a second position when the palm attribute is of the left hand.
 13. The electronic system of claim 12, wherein the electronic system is substantially disposed between a driver's seat and a front passenger's seat of a vehicle, and the first position is in front of a left seat of the vehicle, and the second position is in front of a right seat of the vehicle.
 14. The electronic system of claim 10, wherein the stimulated application corresponds to the user identity is a personalized desktop application on a display device.
 15. A vehicle, comprising: an infotainment system comprising: an image capturing unit configured to capture a plurality of images; a sensor unit configured to detect an object within a sensing region; a display device; a machinery configured to control a movement maneuver of the display device; and a processing unit configured to perform instructions for: recognizing a palm feature from the captured images when the object is detected within the sensing region of the sensor unit; displaying an application on the display device in response to the palm feature, wherein the palm feature includes a palm gesture; determining the movement maneuver of the display device according to directional information of the palm gesture; and moving the display device to a desirable position and a desirable orientation according to the movement maneuver.
 16. The vehicle of claim 15, wherein the processor is further configured to: monitoring a state of a driver in the vehicle; determining a distraction level of the driver according to the state of the driver; and turning off the infotainment system when the distraction level exceeds a threshold.
 17. The vehicle of claim 15, wherein the processor is further configured to: identifying a user identity according to the palm feature; providing a first content when the user identity corresponds to a driver; and providing a second content when the user identity corresponds to a passenger.
 18. The vehicle of claim 15, wherein the display device is moved to a first orientation and a first position in front of a driver seat or a second orientation and a second position in front of a passenger seat. 